Gestures During Overlapping Speech in multimodal Human−Machine Dialogues

A dialogue system has to deal with the problem of interruptions by the user, e.g. changes of requests (called »barge−in«). This contribution is concerned with this problem in the special case of the multimodal dialogue system SmartKom. How are gestures used during such interruptions if they are utilized at all? To answer this question we analyzed a number of human− machine dialogues qualitatively. The analysis showed that most overlap situations were not accompanied by gestures at all. In the remaining instances the gestures were almost never "interactional" gestures, but mostly "unidentifiable" and "emotional" ones. We allocated the overlap situations accompanied by gestures to several subcategories of barge−in, pointed out the peculiarities of the gestures in the different cases and discussed their suitability as indicators for the dialogue system. Although from a small−scale in depth analysis no generalizations can be drawn, valuable insights for further investigation have been won. Most importantly it can be noted that the dynamic features of gestures seem far more promising as indicators of dialogue situations that need to be taken care of by the dialogue system than their static features. 1.1